In [1]:
# GeoTiff images
import rasterio
from osgeo import gdal
# data visualisation
from matplotlib import pyplot as plt
from matplotlib.pyplot import figure
import matplotlib.image as img
from PIL import Image
# model building
import ultralytics
from ultralytics import YOLO
import labelme2yolo
# others
import os
import shutil
import zipfile
In [2]:
# supress warnings (be careful while using this)
import warnings
warnings.filterwarnings('ignore')
# import common GIS tools
import numpy as np
import xarray as xr
import matplotlib.pyplot as plt
import rasterio.features
import rioxarray as rio
from matplotlib.cm import RdYlGn, Reds
# import Planetary Computer tools
import pystac_client
import planetary_computer as pc
import odc
from odc.stac import stac_load
# additional libraries
from datetime import date # date-related calculations
Introduction¶
We undertook a challenge that revolved around the devastating impact of Hurricane Maria, which struck Puerto Rico in 2017. This challenge involved analyzing pre- and post-event satellite images of the entire map of San Juan, Puerto Rico. We used an object detection model that reached a MAP score around 0.75 and helped to identify 4 types of buildings: Damaged Residential, Damaged Commercial, Undamaged Residential, and Undamaged Commercial.
After experimenting with two methods, we opted for using a larger dataset despite its lower precision, prioritizing scalability and a higher MAP score to address widespread impact. The model faced challenges with class imbalance, which negatively affected the detection of damaged buildings. Our findings are applicable in various domains such as infrastructure planning, durable construction, area zoning, insurance, disaster simulations, and historical analyses.
Top Three Actionable Recommendations¶
1. Creating automated algorithms of Machine Learning that will have real-time surface level vision. They would analyze data and provide the fastest possible response to affected areas. Such an approach involves pre-trained data that adds new data and learns from it, providing an edge over other techniques aimed at solving similar problems. Moreover, automation of alerts would eliminate the need for constant human control, reducing maintenance costs and alternative methods. This could become the most responsive tool for such disasters when applying more data and adding real-time information.
2. Enhancing data with alternative sources of imagery like drones, ground-based footage, and even live social media updates. Such an approach will allow for a better identification of affected areas combined with Machine Learning algorithms that are trained on various features. It will provide a deeper understanding of how buildings are damaged from sides or closer distances, allowing for a more detailed examination and assessment of the areas. Bringing multi-source training will expand business uses. Besides that, it will provide more value to citizens' safety as well as the government’s reaction to such events.
3. Based on further dataset analysis and comparison of buildings that were damaged versus undamaged, we propose that particular building structures could be identified as 'stronger'—as fewer of them were destroyed—and 'weaker' structures respectively. This could be helpful for future building initiatives to combat similar events. Some more affected areas could be identified as "more dangerous" and appropriate strategies for habitation there could be created. Such an approach will be helpful to evaluate infrastructure and areas of Puerto Rico that have higher chances of damage. Further, '911' locations could be placed closer, and in similar events, the reaction time to help people would be drastically faster.
Model Analysis + Insights¶
We use the 'Y8x' pre-trained model, which is the largest YOLO-v8 model available. It requires significant resources, therefore, we use 'runpod.io' to rent powerful GPUs for model training. Initially, we used a set of images that had fewer buildings and were relatively easier to classify to establish a baseline for our model. After this initial phase, we expanded our dataset to include a total of 125 images that were densely populated with buildings in various conditions, to increase the complexity and realism of our training data. This total includes both pre-event and post-event pictures, specifically 38 pre-event and 87 post-event images. We identified the area of each specific tile on Google Maps and meticulously went through every single building, cross-checking and verifying each type of building—commercial, residential, damaged, or undamaged. This process revealed the multitude of variables present in each image, highlighting the challenges in labeling. This step was crucial in refining our model’s ability to accurately classify complex urban environments. The baseline model showed a 0.38 MAP score. To improve the result, we applied some data augmentation techniques:
1) Flipping images upside down and from left to right. By that the dataset diversity is increased and its reasonable transformation as buildings orientation does not matter and can be any when satellite photo is made.
2) Increasing saturation/brightness. That helps the model to better recognize objects. As we do not classify things based on saturation/brightness it’s a reasonable transformation.
3) Custom rotation. (Same as #1)
4) Zoom and scale. (Same as #1)
5) Image combination. We combine from 2 to 4 images into one creating a composite image. Enhances the model's ability to generalize by introducing label noise and visual variability.
6)Copying objects from one image and pasting them onto another, useful for increasing object instances and learning object occlusion.
All of the above increased the score up to 0.45. We can see that object detection models require a much larger dataset to be trained. So, we switched to a pre-labeled dataset on Roboflow consisting of thousands of tiles already labeled. We gave up some quality of labeling to gain a larger dataset and this trade-off turned out to be valuable as we achieved a MAP score around 0.75. It was enough to make 100 epochs with a batch-size of 32 without augmentation to achieve such a score. However, we expect that the final evaluation result will be affected by the bias of labeling for this dataset and as a prospect for improvement, we would label thousands of images manually, taking into account information from Google Maps and additional sources to achieve good precision in labeling.
Additional Insights:¶
- The city of Puerto Rico had many damaged buildings, some of which were already in disrepair before the hurricane.
- Upon analyzing Google Maps, it was noted that many commercial buildings are located on road curves, likely because this provides greater visibility and easier access.
- Another observation was that there were many areas where residential and commercial buildings alternated, both having the same size and color. Without the use of Google Maps, it was difficult to distinguish them.
- There are residential blocks where lines of residential houses have walls parallel and adjacent to lines of commercial houses (of the same size). This might indicate that in some areas, there is a preference to have all commercial premises on one side. This complicates the identification of commercial buildings from residential ones without Google Maps.
- Typically, residential areas had red roofs, though not consistently so, and both building types sometimes had white or gray roofs, complicating identification.
- Errors were found in the dataset where a roof of a building was expected, but instead, it was a courtyard, parking lot, or well-maintained driveways. The use of Google Maps helped to understand these discrepancies.
- Some roofs in the images post-event appeared damaged or different from before, but this was not necessarily an indicator of a destroyed building. Higher-definition images showed that these were branches or leaves from trees destroyed by the event.
- A recurring pattern found in buildings destroyed by the hurricane was the dark brown/orange color of some parts of the roofs (not bright colors), which was due to the roofs losing some parts, leaving parts of the frame visible. Many other roofs might have appeared destroyed, but actually, they were buildings already damaged before the hurricane.
Conclusion¶
We identified effective strategies to enhance ML algorithm performance, balancing time and budget constraints. Ideally, integrating human labeling with Google Maps would refine accuracy, requiring analysis of thousands of images for optimal results. This approach, though less precise initially, can improve as the model labels new data, maximizing long-term efficiency. We recommend this development strategy to extend reach and minimize costs. Implementing this algorithm could greatly enhance the quality of life in Puerto Rico by providing sturdier infrastructure, improved building designs, and quicker emergency responses.
Our team would implement/improve if we were given three months to work on this project¶
Our team had a few main challenges which are related to classification of objects and scaling labels throughout a larger dataset. If we had three more months, we would create an automated Machine Learning algorithm that would identify building types and their “damage” status based on the 3 parameters: pre- and post-event images from the satellite and Google Maps (or a substitute). First, the model would identify which buildings are residential and commercial, which would work based on the pre-event imagery from the satellite and aligning it with Google Maps. It would provide the highest precision and lower cost. Further, the outcome - labeled images - would be aligned with the post-event image to identify the buildings affected by the hurricane.
This algorithm would use supervised learning (based on Google Maps) and further create a best strategy to identify classes/labels needed for the object detection. This output could be implemented for similar challenges and utilized not only to identify buildings affected by hurricanes but also improve ML algorithms that are utilized in military, insurance for houses, and building conditions. It could be spinned off to identify which buildings had the highest “survival rate” and create similar structures for a better climate resistance.
Consideration¶
*(The following parts are not included in the word count)
We found the challenge of applying real object detection techniques to a concrete case immensely educational and rewarding. It was extremely interesting to discover how our work on distinguishing between damaged and undamaged buildings could significantly benefit sectors such as insurance and disaster response teams. This potential to enable swift and informed actions in the aftermath of disasters added a meaningful and practical dimension to our learning experience. Additionally, the opportunity to have a call with an expert from NASA was invaluable. It deepened our understanding and fuelled our enthusiasm to apply these technologies in real-world scenarios.
Area for Improvement¶
While the challenge was highly instructive, we believe there is room for improvement in defining the criteria for what constitutes 'damage' in the training models. Refining these definitions could lead to more accurate assessments, which is critical for all subsequent decision-making processes. Moreover, incorporating a robust user feedback mechanism would significantly enhance the adaptability and reliability of the models. This would allow for ongoing improvements based on real-world application and feedback, ensuring that the models remain relevant and effective under varied and changing conditions. Further, enhancing the dataset with more diverse examples of damage from different disasters around the world could provide broader learning opportunities and improve the model's generalizability, making it more versatile and applicable across different geographic and environmental contexts. These enhancements are crucial for maximizing the utility and effectiveness of the models in diverse and dynamic disaster management settings.
References:¶
Artefact. (n.d.). Applying machine learning algorithms to satellite imagery for agriculture applications. Artefact. Retrieved [ 22 April, 2024], from https://www.artefact.com/blog/applying-machine-learning-algorithms-to-satellite-imagery-for-agriculture-applications/
LaTourrette, T., Miller, B. M., Ulin, T., Van Abel, K., Langeland, K. S., & Nanda, N. (n.d.). Public buildings in Puerto Rico after Hurricane Maria: Prestorm challenges, hurricane damage, and suggested courses of action for recovery. Homeland Security Operational Analysis Center (HSOAC), RAND Corporation. Retrieved from https://www.preventionweb.net/publication/public-buildings-puerto-rico-after-hurricane-maria-prestorm-challenges-hurricane-damage
National Aeronautics and Space Administration. (2024). How does a hurricane form? SciJinks. https://scijinks.gov/hurricane/
Puerto Rico Building Code. (2018). Adoption info: Effective dates November 15, 2018 - Present. Adopts with amendments International Building Code 2018 (IBC 2018). Retrieved from https://up.codes/viewer/puerto_rico/ibc-2018
Viasat. (n.d.). How satellites play a role in disaster relief. Viasat Corporate Blog. Retrieved [ 20 April, 2024], from https://news.viasat.com/blog/corporate/how-satellites-play-a-role-in-disaster-relief
Weather Forecast Office. (2017). Major Hurricane Maria - September 20, 2017. Retrieved from https://www.weather.gov/sju/major_hurricane_maria
Weather Forecast Office. (2017). Hurricane Irma - September 5-7, 2017. Retrieved from https://www.weather.gov/sju/hurricane_irma
Getting Data¶
In this part we download satellite data for Puerto Rico coordinates. Time period covers both pre-hurricane and post-hurricane dates.
We display some of the examples below
In [3]:
## Hurricane Maria - San Juan, Puerto Rico ##
# Define the bounding box for the entire data region
min_lon = -66.19385887
min_lat = 18.27306794
max_lon = -66.08007533
max_lat = 18.48024350
# setting geographic boundary
bounds = (min_lon, min_lat, max_lon, max_lat)
# setting time window
time_window = "2017-01-01/2018-01-01"
In [4]:
# connecting to the planetary computer
stac = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")
# seaching for data
search = stac.search(collections = ["sentinel-2-l2a"],
bbox = bounds,
datetime = time_window)
# instantiating results list
items = list(search.get_all_items())
# summarizing results
print('This is the number of scenes that touch our region:',len(items))
This is the number of scenes that touch our region: 101
In [5]:
# checking individual results
for item in items[0:5]:
print(item)
<Item id=S2B_MSIL2A_20171229T150709_R082_T19QHA_20201014T171652> <Item id=S2B_MSIL2A_20171229T150709_R082_T19QGA_20201014T171649> <Item id=S2A_MSIL2A_20171224T150711_R082_T19QHA_20201014T160213> <Item id=S2A_MSIL2A_20171224T150711_R082_T19QGA_20201014T160212> <Item id=S2B_MSIL2A_20171219T150709_R082_T19QHA_20201014T151203>
In [6]:
# pixel resolution for the final product
resolution = 10 # meters per pixel
# scaling to degrees per pizel
scale = resolution / 111320.0 # degrees per pixel for CRS:4326
In [7]:
xx = stac_load(
items,
bands = ["red", "green", "blue", "nir", "SCL"],
crs = "EPSG:4326", # latitude-longitude
resolution = scale, # degrees
chunks = {"x": 2048, "y": 2048},
dtype = "uint16",
patch_url = pc.sign,
bbox = bounds
)
In [8]:
# view the dimensions of our XARRAY and the variables
display(xx)
<xarray.Dataset>
Dimensions: (latitude: 2308, longitude: 1268, time: 52)
Coordinates:
* latitude (latitude) float64 18.48 18.48 18.48 ... 18.27 18.27 18.27
* longitude (longitude) float64 -66.19 -66.19 -66.19 ... -66.08 -66.08
spatial_ref int32 4326
* time (time) datetime64[ns] 2017-01-08T15:07:11.026000 ... 2017-12...
Data variables:
red (time, latitude, longitude) uint16 dask.array<chunksize=(1, 2048, 1268), meta=np.ndarray>
green (time, latitude, longitude) uint16 dask.array<chunksize=(1, 2048, 1268), meta=np.ndarray>
blue (time, latitude, longitude) uint16 dask.array<chunksize=(1, 2048, 1268), meta=np.ndarray>
nir (time, latitude, longitude) uint16 dask.array<chunksize=(1, 2048, 1268), meta=np.ndarray>
SCL (time, latitude, longitude) uint16 dask.array<chunksize=(1, 2048, 1268), meta=np.ndarray>In [9]:
## This code will take some time to process ##
# subsetting results for RGB
plot_xx = xx[ ["red", "green", "blue"] ].to_array()
# showing results
plot_xx.plot.imshow(col = 'time', # time
col_wrap = 4 , # four columns per row
robust = True ,
vmin = 0 ,
vmax = 3000 )
# rendering results
plt.show()
In [16]:
# select a time slice to view a single RGB image and the cloud mask
# see the xarray dimensions above for the number of time slices (starts at 0)
time_slice = 47 # December 9, 2017 (post-storm)
In [17]:
import matplotlib.pyplot as plt
In [18]:
## plotting an RGB real color image for a single date ##
# setting plot size
fig, ax = plt.subplots( figsize = (6, 10) )
# preparing the plot
xx.isel(time = time_slice)[ ["red", "green", "blue"] ].\
to_array().plot.imshow(robust = True,
ax = ax ,
vmin = 0 ,
vmax = 3000)
# titles and axis lables
ax.set_title(label = f"RGB Color Results for Scene {time_slice}")
ax.axis('off')
# rendering results
plt.show()
Images analysis¶
We colorize images recognizing noisy elements such as clouds, water etc.
Afterwards we measure vegetation level (NDVI) to highlight buildings and to compare pre-event state with post-event state. It will be used to recognize the area having most dramatic change to analyze it further.
In [19]:
# instantiating a colormap for SCL pixel classifications
scl_colormap = np.array(
[
[252, 40, 228, 255], # 0 - NODATA - MAGENTA
[255, 0, 4, 255], # 1 - Saturated or Defective - RED
[0 , 0, 0, 255], # 2 - Dark Areas - BLACK
[97 , 97, 97, 255], # 3 - Cloud Shadow - DARK GREY
[3 , 139, 80, 255], # 4 - Vegetation - GREEN
[192, 132, 12, 255], # 5 - Bare Ground - BROWN
[21 , 103, 141, 255], # 6 - Water - BLUE
[117, 0, 27, 255], # 7 - Unclassified - MAROON
[208, 208, 208, 255], # 8 - Cloud - LIGHT GREY
[244, 244, 244, 255], # 9 - Definitely Cloud - WHITE
[195, 231, 240, 255], # 10 - Thin Cloud - LIGHT BLUE
[222, 157, 204, 255], # 11 - Snow or Ice - PINK
],
dtype="uint8",
)
In [20]:
# function for color encoding
def colorize(xx, colormap):
return xr.DataArray( colormap[xx.data],
coords = xx.coords,
dims = (*xx.dims, "band") )
In [21]:
# filtering out water, etc.
filter_values = [0, 1, 3, 6, 8, 9, 10]
cloud_mask = ~xx.SCL.isin(filter_values) # this means not in filter_values
In [22]:
# appling cloud mask (filtering out clouds, cloud shadows, and water)
# storing as 16-bit integers
cleaned_data = xx.where(cloud_mask).astype("uint16")
In [23]:
# converting SCL to RGB
scl_rgba_clean = colorize(xx = cleaned_data.isel(time = time_slice).SCL.compute(),
colormap = scl_colormap)
# setting figure size
plt.figure(figsize = (6, 10))
plt.imshow(scl_rgba_clean)
# titles and axis labels
plt.title(label = "Cloud / Shadows / Water Mask (MAGENTA)")
plt.axis('off')
# rendering the plot
plt.show()
In [32]:
# preparing two time steps compare NDVI outputs
first_time = 3 #pre
second_time = 47 #post
first_time_1 = 8 #pre
second_time_1 = 46 #post
In [40]:
# Plots of NDVI at two different time slices
# Clouds, Shadows and Water are masked
# setting figure size
fig, ax = plt.subplots(2, 2, figsize = (20, 15))
#prepare images
ndvi_image = (cleaned_data.nir - cleaned_data.red) / (cleaned_data.nir + cleaned_data.red)
# first image data
ndvi_image.isel(time = first_time).plot(ax = ax[0][0],
vmin = 0.0,
vmax = 0.8,
cmap = "RdYlGn")
# second image data
ndvi_image.isel(time = second_time).plot(ax = ax[0][1],
vmin = 0.0,
vmax = 0.8,
cmap = "RdYlGn")
# 3rd image data
ndvi_image.isel(time = first_time_1 ).plot(ax = ax[1][0],
vmin = 0.0,
vmax = 0.8,
cmap = "RdYlGn")
# 4th image data
ndvi_image.isel(time = second_time_1 ).plot(ax = ax[1][1],
vmin = 0.0,
vmax = 0.8,
cmap = "RdYlGn")
# axis labels
ax[0][0].set_title(label = 'NDVI-Time Pre #1'), ax[0][0].xaxis.set_visible(False), ax[0][0].yaxis.set_visible(False)
ax[0][1].set_title(label = 'NDVI-Time Post #1'), ax[0][1].xaxis.set_visible(False), ax[0][1].yaxis.set_visible(False)
ax[1][0].set_title(label = 'NDVI-Time Pre #2'), ax[1][0].xaxis.set_visible(False), ax[1][0].yaxis.set_visible(False)
ax[1][1].set_title(label = 'NDVI-Time Post #2'), ax[1][1].xaxis.set_visible(False), ax[1][1].yaxis.set_visible(False)
# rendering the plot
plt.show()
NDVI Anomalies¶
Here we inspect differences in vegetation level throughout all land areas. Some parts of the image could be covered with clouds thus we use 3 different combinations of pre and post photos to see the whole are as a result. In places with highest color differences we expect to see significant damage to the buildings.
In [36]:
# function for calculating NDVI anomalies
def NDVI(dataset):
return (dataset.nir - dataset.red) / (dataset.nir + dataset.red)
In [37]:
# running comparison
ndvi_clean = NDVI(cleaned_data)
In [41]:
# calculating difference for 3 and 47
ndvi_pre = ndvi_clean.isel(time = first_time)
ndvi_post = ndvi_clean.isel(time = second_time)
ndvi_anomaly = ndvi_post - ndvi_pre
# calculating difference for 8 and 46
ndvi_pre_1 = ndvi_clean.isel(time = first_time_1)
ndvi_post_1 = ndvi_clean.isel(time = second_time_1)
ndvi_anomaly_1 = ndvi_post_1 - ndvi_pre_1
# calculating difference for 8 and 47
ndvi_anomaly_2 = ndvi_post - ndvi_pre_1
# all areas of water or clouds will be black
RdYlGn.set_bad('black',1.)
# reversing the colormap for reds
Reds_reverse = "Reds_r"
In [43]:
## plotting NDVI anomaly
plt.figure( figsize = (6,10) )
ndvi_anomaly.plot(vmin = -0.2, vmax=0.0, cmap = Reds_reverse, add_colorbar=False)
# titles and labels
plt.title (label = "NDVI Anomaly 0")
plt.xlabel(xlabel = "Longitude")
plt.ylabel(ylabel = "Latitude")
# rendering the plot
plt.show()
In [44]:
## plotting NDVI anomaly
plt.figure( figsize = (6,10) )
ndvi_anomaly_1.plot(vmin = -0.2, vmax=0.0, cmap = Reds_reverse, add_colorbar=False)
# titles and labels
plt.title (label = "NDVI Anomaly 1")
plt.xlabel(xlabel = "Longitude")
plt.ylabel(ylabel = "Latitude")
# rendering the plot
plt.show()
In [45]:
## plotting NDVI anomaly
plt.figure( figsize = (6,10) )
ndvi_anomaly_2.plot(vmin = -0.2, vmax=0.0, cmap = Reds_reverse, add_colorbar=False)
# titles and labels
plt.title (label = "NDVI Anomaly 2")
plt.xlabel(xlabel = "Longitude")
plt.ylabel(ylabel = "Latitude")
# rendering the plot
plt.show()
Hurricanes work in such a way where they have ‘calm eye’ in the middle, followed by a stronger wall of winds which extends with heavy rains to thousands of kilometers. They also become weaker at the coller water and once they get further from water. So we noticed some areas being particularly damaged although some buildings looked untouched. Beside that, we noticed that the most impact from the hurricane happened in the poorer areas. The materials of the houses weren't able to withstand the impact and on most images we can see pieces of roof materials on the ground beside buildings which indicates that this rationale works. (NASA, 2024)
Model Creation¶
We upload 2 pre-trained models which we were training on GPUs (Those were rented at runpod.io).
(In the comments we left the code we used for training and data downloading)
In the end we show results and performance of the best model, which is used to make final predictions as well.
In [47]:
#model trained on huge dataset
model = YOLO('/Users/macbookair13m22022/Downloads/train11/weights/best.pt')
#model trained on our manually-labeled dataset
model_2 = YOLO('/Users/macbookair13m22022/Downloads/train5222/weights/best.pt')
In [48]:
model.info()
Model summary: 365 layers, 68156460 parameters, 0 gradients, 258.1 GFLOPs
Out[48]:
(365, 68156460, 0, 258.13867519999997)
In [49]:
model_2.info()
Model summary: 365 layers, 68156460 parameters, 0 gradients, 258.1 GFLOPs
Out[49]:
(365, 68156460, 0, 258.13867519999997)
In [ ]:
#This is how to get our dataset (if needed)
# !pip install roboflow
# from roboflow import Roboflow
# rf = Roboflow(api_key="dnsnQIJiOmaJ6AWlNb8u")
# project = rf.workspace("bciii-new").project("nasa-google-maps")
# version = project.version(2)
# dataset = version.download("yolov8")
In [ ]:
#This is how the model was trained
# Train the model on the dataset for 50 epochs
#results = model.train(data = './Damaged-Buildings-1/data.yaml',
# epochs = 100 ,
# imgsz = 512,
# batch = 32,
# flipud = 0.5,
# fliplr = 0.5,
# hsv_s = 0.8,
# hsv_v = 0.6,
# scale = 1,
# degrees = 180,
# mosaic = 0.5,
# mixup = 0.5,
# copy_paste = 0.5,
# device = 'cuda')
In [50]:
#Use sample dataset to see some part of results (sample dataset has to be downloaded and be in the same folder with this code)
model.val()
Ultralytics YOLOv8.2.2 🚀 Python-3.11.0 torch-2.1.2 CPU (Apple M2) Model summary (fused): 268 layers, 68127420 parameters, 0 gradients, 257.4 GFLOPs
val: Scanning /Users/macbookair13m22022/Downloads/Damaged-Buildings-1/valid/labe
val: New cache created: /Users/macbookair13m22022/Downloads/Damaged-Buildings-1/valid/labels.cache
Class Images Instances Box(P R mAP50 m
all 2 47 0.8 0.918 0.914 0.805
damagedresidentialbuilding 2 13 0.699 0.894 0.846 0.74
undamagedresidentialbuilding 2 34 0.902 0.941 0.982 0.87
Speed: 2.8ms preprocess, 760.7ms inference, 0.0ms loss, 375.1ms postprocess per image
Results saved to runs/detect/val
Out[50]:
ultralytics.utils.metrics.DetMetrics object with attributes:
ap_class_index: array([1, 3])
box: ultralytics.utils.metrics.Metric object
confusion_matrix: <ultralytics.utils.metrics.ConfusionMatrix object at 0x2b59bf610>
curves: ['Precision-Recall(B)', 'F1-Confidence(B)', 'Precision-Confidence(B)', 'Recall-Confidence(B)']
curves_results: [[array([ 0, 0.001001, 0.002002, 0.003003, 0.004004, 0.005005, 0.006006, 0.007007, 0.008008, 0.009009, 0.01001, 0.011011, 0.012012, 0.013013, 0.014014, 0.015015, 0.016016, 0.017017, 0.018018, 0.019019, 0.02002, 0.021021, 0.022022, 0.023023,
0.024024, 0.025025, 0.026026, 0.027027, 0.028028, 0.029029, 0.03003, 0.031031, 0.032032, 0.033033, 0.034034, 0.035035, 0.036036, 0.037037, 0.038038, 0.039039, 0.04004, 0.041041, 0.042042, 0.043043, 0.044044, 0.045045, 0.046046, 0.047047,
0.048048, 0.049049, 0.05005, 0.051051, 0.052052, 0.053053, 0.054054, 0.055055, 0.056056, 0.057057, 0.058058, 0.059059, 0.06006, 0.061061, 0.062062, 0.063063, 0.064064, 0.065065, 0.066066, 0.067067, 0.068068, 0.069069, 0.07007, 0.071071,
0.072072, 0.073073, 0.074074, 0.075075, 0.076076, 0.077077, 0.078078, 0.079079, 0.08008, 0.081081, 0.082082, 0.083083, 0.084084, 0.085085, 0.086086, 0.087087, 0.088088, 0.089089, 0.09009, 0.091091, 0.092092, 0.093093, 0.094094, 0.095095,
0.096096, 0.097097, 0.098098, 0.099099, 0.1001, 0.1011, 0.1021, 0.1031, 0.1041, 0.10511, 0.10611, 0.10711, 0.10811, 0.10911, 0.11011, 0.11111, 0.11211, 0.11311, 0.11411, 0.11512, 0.11612, 0.11712, 0.11812, 0.11912,
0.12012, 0.12112, 0.12212, 0.12312, 0.12412, 0.12513, 0.12613, 0.12713, 0.12813, 0.12913, 0.13013, 0.13113, 0.13213, 0.13313, 0.13413, 0.13514, 0.13614, 0.13714, 0.13814, 0.13914, 0.14014, 0.14114, 0.14214, 0.14314,
0.14414, 0.14515, 0.14615, 0.14715, 0.14815, 0.14915, 0.15015, 0.15115, 0.15215, 0.15315, 0.15415, 0.15516, 0.15616, 0.15716, 0.15816, 0.15916, 0.16016, 0.16116, 0.16216, 0.16316, 0.16416, 0.16517, 0.16617, 0.16717,
0.16817, 0.16917, 0.17017, 0.17117, 0.17217, 0.17317, 0.17417, 0.17518, 0.17618, 0.17718, 0.17818, 0.17918, 0.18018, 0.18118, 0.18218, 0.18318, 0.18418, 0.18519, 0.18619, 0.18719, 0.18819, 0.18919, 0.19019, 0.19119,
0.19219, 0.19319, 0.19419, 0.1952, 0.1962, 0.1972, 0.1982, 0.1992, 0.2002, 0.2012, 0.2022, 0.2032, 0.2042, 0.20521, 0.20621, 0.20721, 0.20821, 0.20921, 0.21021, 0.21121, 0.21221, 0.21321, 0.21421, 0.21522,
0.21622, 0.21722, 0.21822, 0.21922, 0.22022, 0.22122, 0.22222, 0.22322, 0.22422, 0.22523, 0.22623, 0.22723, 0.22823, 0.22923, 0.23023, 0.23123, 0.23223, 0.23323, 0.23423, 0.23524, 0.23624, 0.23724, 0.23824, 0.23924,
0.24024, 0.24124, 0.24224, 0.24324, 0.24424, 0.24525, 0.24625, 0.24725, 0.24825, 0.24925, 0.25025, 0.25125, 0.25225, 0.25325, 0.25425, 0.25526, 0.25626, 0.25726, 0.25826, 0.25926, 0.26026, 0.26126, 0.26226, 0.26326,
0.26426, 0.26527, 0.26627, 0.26727, 0.26827, 0.26927, 0.27027, 0.27127, 0.27227, 0.27327, 0.27427, 0.27528, 0.27628, 0.27728, 0.27828, 0.27928, 0.28028, 0.28128, 0.28228, 0.28328, 0.28428, 0.28529, 0.28629, 0.28729,
0.28829, 0.28929, 0.29029, 0.29129, 0.29229, 0.29329, 0.29429, 0.2953, 0.2963, 0.2973, 0.2983, 0.2993, 0.3003, 0.3013, 0.3023, 0.3033, 0.3043, 0.30531, 0.30631, 0.30731, 0.30831, 0.30931, 0.31031, 0.31131,
0.31231, 0.31331, 0.31431, 0.31532, 0.31632, 0.31732, 0.31832, 0.31932, 0.32032, 0.32132, 0.32232, 0.32332, 0.32432, 0.32533, 0.32633, 0.32733, 0.32833, 0.32933, 0.33033, 0.33133, 0.33233, 0.33333, 0.33433, 0.33534,
0.33634, 0.33734, 0.33834, 0.33934, 0.34034, 0.34134, 0.34234, 0.34334, 0.34434, 0.34535, 0.34635, 0.34735, 0.34835, 0.34935, 0.35035, 0.35135, 0.35235, 0.35335, 0.35435, 0.35536, 0.35636, 0.35736, 0.35836, 0.35936,
0.36036, 0.36136, 0.36236, 0.36336, 0.36436, 0.36537, 0.36637, 0.36737, 0.36837, 0.36937, 0.37037, 0.37137, 0.37237, 0.37337, 0.37437, 0.37538, 0.37638, 0.37738, 0.37838, 0.37938, 0.38038, 0.38138, 0.38238, 0.38338,
0.38438, 0.38539, 0.38639, 0.38739, 0.38839, 0.38939, 0.39039, 0.39139, 0.39239, 0.39339, 0.39439, 0.3954, 0.3964, 0.3974, 0.3984, 0.3994, 0.4004, 0.4014, 0.4024, 0.4034, 0.4044, 0.40541, 0.40641, 0.40741,
0.40841, 0.40941, 0.41041, 0.41141, 0.41241, 0.41341, 0.41441, 0.41542, 0.41642, 0.41742, 0.41842, 0.41942, 0.42042, 0.42142, 0.42242, 0.42342, 0.42442, 0.42543, 0.42643, 0.42743, 0.42843, 0.42943, 0.43043, 0.43143,
0.43243, 0.43343, 0.43443, 0.43544, 0.43644, 0.43744, 0.43844, 0.43944, 0.44044, 0.44144, 0.44244, 0.44344, 0.44444, 0.44545, 0.44645, 0.44745, 0.44845, 0.44945, 0.45045, 0.45145, 0.45245, 0.45345, 0.45445, 0.45546,
0.45646, 0.45746, 0.45846, 0.45946, 0.46046, 0.46146, 0.46246, 0.46346, 0.46446, 0.46547, 0.46647, 0.46747, 0.46847, 0.46947, 0.47047, 0.47147, 0.47247, 0.47347, 0.47447, 0.47548, 0.47648, 0.47748, 0.47848, 0.47948,
0.48048, 0.48148, 0.48248, 0.48348, 0.48448, 0.48549, 0.48649, 0.48749, 0.48849, 0.48949, 0.49049, 0.49149, 0.49249, 0.49349, 0.49449, 0.4955, 0.4965, 0.4975, 0.4985, 0.4995, 0.5005, 0.5015, 0.5025, 0.5035,
0.5045, 0.50551, 0.50651, 0.50751, 0.50851, 0.50951, 0.51051, 0.51151, 0.51251, 0.51351, 0.51451, 0.51552, 0.51652, 0.51752, 0.51852, 0.51952, 0.52052, 0.52152, 0.52252, 0.52352, 0.52452, 0.52553, 0.52653, 0.52753,
0.52853, 0.52953, 0.53053, 0.53153, 0.53253, 0.53353, 0.53453, 0.53554, 0.53654, 0.53754, 0.53854, 0.53954, 0.54054, 0.54154, 0.54254, 0.54354, 0.54454, 0.54555, 0.54655, 0.54755, 0.54855, 0.54955, 0.55055, 0.55155,
0.55255, 0.55355, 0.55455, 0.55556, 0.55656, 0.55756, 0.55856, 0.55956, 0.56056, 0.56156, 0.56256, 0.56356, 0.56456, 0.56557, 0.56657, 0.56757, 0.56857, 0.56957, 0.57057, 0.57157, 0.57257, 0.57357, 0.57457, 0.57558,
0.57658, 0.57758, 0.57858, 0.57958, 0.58058, 0.58158, 0.58258, 0.58358, 0.58458, 0.58559, 0.58659, 0.58759, 0.58859, 0.58959, 0.59059, 0.59159, 0.59259, 0.59359, 0.59459, 0.5956, 0.5966, 0.5976, 0.5986, 0.5996,
0.6006, 0.6016, 0.6026, 0.6036, 0.6046, 0.60561, 0.60661, 0.60761, 0.60861, 0.60961, 0.61061, 0.61161, 0.61261, 0.61361, 0.61461, 0.61562, 0.61662, 0.61762, 0.61862, 0.61962, 0.62062, 0.62162, 0.62262, 0.62362,
0.62462, 0.62563, 0.62663, 0.62763, 0.62863, 0.62963, 0.63063, 0.63163, 0.63263, 0.63363, 0.63463, 0.63564, 0.63664, 0.63764, 0.63864, 0.63964, 0.64064, 0.64164, 0.64264, 0.64364, 0.64464, 0.64565, 0.64665, 0.64765,
0.64865, 0.64965, 0.65065, 0.65165, 0.65265, 0.65365, 0.65465, 0.65566, 0.65666, 0.65766, 0.65866, 0.65966, 0.66066, 0.66166, 0.66266, 0.66366, 0.66466, 0.66567, 0.66667, 0.66767, 0.66867, 0.66967, 0.67067, 0.67167,
0.67267, 0.67367, 0.67467, 0.67568, 0.67668, 0.67768, 0.67868, 0.67968, 0.68068, 0.68168, 0.68268, 0.68368, 0.68468, 0.68569, 0.68669, 0.68769, 0.68869, 0.68969, 0.69069, 0.69169, 0.69269, 0.69369, 0.69469, 0.6957,
0.6967, 0.6977, 0.6987, 0.6997, 0.7007, 0.7017, 0.7027, 0.7037, 0.7047, 0.70571, 0.70671, 0.70771, 0.70871, 0.70971, 0.71071, 0.71171, 0.71271, 0.71371, 0.71471, 0.71572, 0.71672, 0.71772, 0.71872, 0.71972,
0.72072, 0.72172, 0.72272, 0.72372, 0.72472, 0.72573, 0.72673, 0.72773, 0.72873, 0.72973, 0.73073, 0.73173, 0.73273, 0.73373, 0.73473, 0.73574, 0.73674, 0.73774, 0.73874, 0.73974, 0.74074, 0.74174, 0.74274, 0.74374,
0.74474, 0.74575, 0.74675, 0.74775, 0.74875, 0.74975, 0.75075, 0.75175, 0.75275, 0.75375, 0.75475, 0.75576, 0.75676, 0.75776, 0.75876, 0.75976, 0.76076, 0.76176, 0.76276, 0.76376, 0.76476, 0.76577, 0.76677, 0.76777,
0.76877, 0.76977, 0.77077, 0.77177, 0.77277, 0.77377, 0.77477, 0.77578, 0.77678, 0.77778, 0.77878, 0.77978, 0.78078, 0.78178, 0.78278, 0.78378, 0.78478, 0.78579, 0.78679, 0.78779, 0.78879, 0.78979, 0.79079, 0.79179,
0.79279, 0.79379, 0.79479, 0.7958, 0.7968, 0.7978, 0.7988, 0.7998, 0.8008, 0.8018, 0.8028, 0.8038, 0.8048, 0.80581, 0.80681, 0.80781, 0.80881, 0.80981, 0.81081, 0.81181, 0.81281, 0.81381, 0.81481, 0.81582,
0.81682, 0.81782, 0.81882, 0.81982, 0.82082, 0.82182, 0.82282, 0.82382, 0.82482, 0.82583, 0.82683, 0.82783, 0.82883, 0.82983, 0.83083, 0.83183, 0.83283, 0.83383, 0.83483, 0.83584, 0.83684, 0.83784, 0.83884, 0.83984,
0.84084, 0.84184, 0.84284, 0.84384, 0.84484, 0.84585, 0.84685, 0.84785, 0.84885, 0.84985, 0.85085, 0.85185, 0.85285, 0.85385, 0.85485, 0.85586, 0.85686, 0.85786, 0.85886, 0.85986, 0.86086, 0.86186, 0.86286, 0.86386,
0.86486, 0.86587, 0.86687, 0.86787, 0.86887, 0.86987, 0.87087, 0.87187, 0.87287, 0.87387, 0.87487, 0.87588, 0.87688, 0.87788, 0.87888, 0.87988, 0.88088, 0.88188, 0.88288, 0.88388, 0.88488, 0.88589, 0.88689, 0.88789,
0.88889, 0.88989, 0.89089, 0.89189, 0.89289, 0.89389, 0.89489, 0.8959, 0.8969, 0.8979, 0.8989, 0.8999, 0.9009, 0.9019, 0.9029, 0.9039, 0.9049, 0.90591, 0.90691, 0.90791, 0.90891, 0.90991, 0.91091, 0.91191,
0.91291, 0.91391, 0.91491, 0.91592, 0.91692, 0.91792, 0.91892, 0.91992, 0.92092, 0.92192, 0.92292, 0.92392, 0.92492, 0.92593, 0.92693, 0.92793, 0.92893, 0.92993, 0.93093, 0.93193, 0.93293, 0.93393, 0.93493, 0.93594,
0.93694, 0.93794, 0.93894, 0.93994, 0.94094, 0.94194, 0.94294, 0.94394, 0.94494, 0.94595, 0.94695, 0.94795, 0.94895, 0.94995, 0.95095, 0.95195, 0.95295, 0.95395, 0.95495, 0.95596, 0.95696, 0.95796, 0.95896, 0.95996,
0.96096, 0.96196, 0.96296, 0.96396, 0.96496, 0.96597, 0.96697, 0.96797, 0.96897, 0.96997, 0.97097, 0.97197, 0.97297, 0.97397, 0.97497, 0.97598, 0.97698, 0.97798, 0.97898, 0.97998, 0.98098, 0.98198, 0.98298, 0.98398,
0.98498, 0.98599, 0.98699, 0.98799, 0.98899, 0.98999, 0.99099, 0.99199, 0.99299, 0.99399, 0.99499, 0.996, 0.997, 0.998, 0.999, 1]), array([[ 1, 1, 1, ..., 0.65, 0.65, 0],
[ 1, 1, 1, ..., 0.73913, 0.73913, 0]]), 'Recall', 'Precision'], [array([ 0, 0.001001, 0.002002, 0.003003, 0.004004, 0.005005, 0.006006, 0.007007, 0.008008, 0.009009, 0.01001, 0.011011, 0.012012, 0.013013, 0.014014, 0.015015, 0.016016, 0.017017, 0.018018, 0.019019, 0.02002, 0.021021, 0.022022, 0.023023,
0.024024, 0.025025, 0.026026, 0.027027, 0.028028, 0.029029, 0.03003, 0.031031, 0.032032, 0.033033, 0.034034, 0.035035, 0.036036, 0.037037, 0.038038, 0.039039, 0.04004, 0.041041, 0.042042, 0.043043, 0.044044, 0.045045, 0.046046, 0.047047,
0.048048, 0.049049, 0.05005, 0.051051, 0.052052, 0.053053, 0.054054, 0.055055, 0.056056, 0.057057, 0.058058, 0.059059, 0.06006, 0.061061, 0.062062, 0.063063, 0.064064, 0.065065, 0.066066, 0.067067, 0.068068, 0.069069, 0.07007, 0.071071,
0.072072, 0.073073, 0.074074, 0.075075, 0.076076, 0.077077, 0.078078, 0.079079, 0.08008, 0.081081, 0.082082, 0.083083, 0.084084, 0.085085, 0.086086, 0.087087, 0.088088, 0.089089, 0.09009, 0.091091, 0.092092, 0.093093, 0.094094, 0.095095,
0.096096, 0.097097, 0.098098, 0.099099, 0.1001, 0.1011, 0.1021, 0.1031, 0.1041, 0.10511, 0.10611, 0.10711, 0.10811, 0.10911, 0.11011, 0.11111, 0.11211, 0.11311, 0.11411, 0.11512, 0.11612, 0.11712, 0.11812, 0.11912,
0.12012, 0.12112, 0.12212, 0.12312, 0.12412, 0.12513, 0.12613, 0.12713, 0.12813, 0.12913, 0.13013, 0.13113, 0.13213, 0.13313, 0.13413, 0.13514, 0.13614, 0.13714, 0.13814, 0.13914, 0.14014, 0.14114, 0.14214, 0.14314,
0.14414, 0.14515, 0.14615, 0.14715, 0.14815, 0.14915, 0.15015, 0.15115, 0.15215, 0.15315, 0.15415, 0.15516, 0.15616, 0.15716, 0.15816, 0.15916, 0.16016, 0.16116, 0.16216, 0.16316, 0.16416, 0.16517, 0.16617, 0.16717,
0.16817, 0.16917, 0.17017, 0.17117, 0.17217, 0.17317, 0.17417, 0.17518, 0.17618, 0.17718, 0.17818, 0.17918, 0.18018, 0.18118, 0.18218, 0.18318, 0.18418, 0.18519, 0.18619, 0.18719, 0.18819, 0.18919, 0.19019, 0.19119,
0.19219, 0.19319, 0.19419, 0.1952, 0.1962, 0.1972, 0.1982, 0.1992, 0.2002, 0.2012, 0.2022, 0.2032, 0.2042, 0.20521, 0.20621, 0.20721, 0.20821, 0.20921, 0.21021, 0.21121, 0.21221, 0.21321, 0.21421, 0.21522,
0.21622, 0.21722, 0.21822, 0.21922, 0.22022, 0.22122, 0.22222, 0.22322, 0.22422, 0.22523, 0.22623, 0.22723, 0.22823, 0.22923, 0.23023, 0.23123, 0.23223, 0.23323, 0.23423, 0.23524, 0.23624, 0.23724, 0.23824, 0.23924,
0.24024, 0.24124, 0.24224, 0.24324, 0.24424, 0.24525, 0.24625, 0.24725, 0.24825, 0.24925, 0.25025, 0.25125, 0.25225, 0.25325, 0.25425, 0.25526, 0.25626, 0.25726, 0.25826, 0.25926, 0.26026, 0.26126, 0.26226, 0.26326,
0.26426, 0.26527, 0.26627, 0.26727, 0.26827, 0.26927, 0.27027, 0.27127, 0.27227, 0.27327, 0.27427, 0.27528, 0.27628, 0.27728, 0.27828, 0.27928, 0.28028, 0.28128, 0.28228, 0.28328, 0.28428, 0.28529, 0.28629, 0.28729,
0.28829, 0.28929, 0.29029, 0.29129, 0.29229, 0.29329, 0.29429, 0.2953, 0.2963, 0.2973, 0.2983, 0.2993, 0.3003, 0.3013, 0.3023, 0.3033, 0.3043, 0.30531, 0.30631, 0.30731, 0.30831, 0.30931, 0.31031, 0.31131,
0.31231, 0.31331, 0.31431, 0.31532, 0.31632, 0.31732, 0.31832, 0.31932, 0.32032, 0.32132, 0.32232, 0.32332, 0.32432, 0.32533, 0.32633, 0.32733, 0.32833, 0.32933, 0.33033, 0.33133, 0.33233, 0.33333, 0.33433, 0.33534,
0.33634, 0.33734, 0.33834, 0.33934, 0.34034, 0.34134, 0.34234, 0.34334, 0.34434, 0.34535, 0.34635, 0.34735, 0.34835, 0.34935, 0.35035, 0.35135, 0.35235, 0.35335, 0.35435, 0.35536, 0.35636, 0.35736, 0.35836, 0.35936,
0.36036, 0.36136, 0.36236, 0.36336, 0.36436, 0.36537, 0.36637, 0.36737, 0.36837, 0.36937, 0.37037, 0.37137, 0.37237, 0.37337, 0.37437, 0.37538, 0.37638, 0.37738, 0.37838, 0.37938, 0.38038, 0.38138, 0.38238, 0.38338,
0.38438, 0.38539, 0.38639, 0.38739, 0.38839, 0.38939, 0.39039, 0.39139, 0.39239, 0.39339, 0.39439, 0.3954, 0.3964, 0.3974, 0.3984, 0.3994, 0.4004, 0.4014, 0.4024, 0.4034, 0.4044, 0.40541, 0.40641, 0.40741,
0.40841, 0.40941, 0.41041, 0.41141, 0.41241, 0.41341, 0.41441, 0.41542, 0.41642, 0.41742, 0.41842, 0.41942, 0.42042, 0.42142, 0.42242, 0.42342, 0.42442, 0.42543, 0.42643, 0.42743, 0.42843, 0.42943, 0.43043, 0.43143,
0.43243, 0.43343, 0.43443, 0.43544, 0.43644, 0.43744, 0.43844, 0.43944, 0.44044, 0.44144, 0.44244, 0.44344, 0.44444, 0.44545, 0.44645, 0.44745, 0.44845, 0.44945, 0.45045, 0.45145, 0.45245, 0.45345, 0.45445, 0.45546,
0.45646, 0.45746, 0.45846, 0.45946, 0.46046, 0.46146, 0.46246, 0.46346, 0.46446, 0.46547, 0.46647, 0.46747, 0.46847, 0.46947, 0.47047, 0.47147, 0.47247, 0.47347, 0.47447, 0.47548, 0.47648, 0.47748, 0.47848, 0.47948,
0.48048, 0.48148, 0.48248, 0.48348, 0.48448, 0.48549, 0.48649, 0.48749, 0.48849, 0.48949, 0.49049, 0.49149, 0.49249, 0.49349, 0.49449, 0.4955, 0.4965, 0.4975, 0.4985, 0.4995, 0.5005, 0.5015, 0.5025, 0.5035,
0.5045, 0.50551, 0.50651, 0.50751, 0.50851, 0.50951, 0.51051, 0.51151, 0.51251, 0.51351, 0.51451, 0.51552, 0.51652, 0.51752, 0.51852, 0.51952, 0.52052, 0.52152, 0.52252, 0.52352, 0.52452, 0.52553, 0.52653, 0.52753,
0.52853, 0.52953, 0.53053, 0.53153, 0.53253, 0.53353, 0.53453, 0.53554, 0.53654, 0.53754, 0.53854, 0.53954, 0.54054, 0.54154, 0.54254, 0.54354, 0.54454, 0.54555, 0.54655, 0.54755, 0.54855, 0.54955, 0.55055, 0.55155,
0.55255, 0.55355, 0.55455, 0.55556, 0.55656, 0.55756, 0.55856, 0.55956, 0.56056, 0.56156, 0.56256, 0.56356, 0.56456, 0.56557, 0.56657, 0.56757, 0.56857, 0.56957, 0.57057, 0.57157, 0.57257, 0.57357, 0.57457, 0.57558,
0.57658, 0.57758, 0.57858, 0.57958, 0.58058, 0.58158, 0.58258, 0.58358, 0.58458, 0.58559, 0.58659, 0.58759, 0.58859, 0.58959, 0.59059, 0.59159, 0.59259, 0.59359, 0.59459, 0.5956, 0.5966, 0.5976, 0.5986, 0.5996,
0.6006, 0.6016, 0.6026, 0.6036, 0.6046, 0.60561, 0.60661, 0.60761, 0.60861, 0.60961, 0.61061, 0.61161, 0.61261, 0.61361, 0.61461, 0.61562, 0.61662, 0.61762, 0.61862, 0.61962, 0.62062, 0.62162, 0.62262, 0.62362,
0.62462, 0.62563, 0.62663, 0.62763, 0.62863, 0.62963, 0.63063, 0.63163, 0.63263, 0.63363, 0.63463, 0.63564, 0.63664, 0.63764, 0.63864, 0.63964, 0.64064, 0.64164, 0.64264, 0.64364, 0.64464, 0.64565, 0.64665, 0.64765,
0.64865, 0.64965, 0.65065, 0.65165, 0.65265, 0.65365, 0.65465, 0.65566, 0.65666, 0.65766, 0.65866, 0.65966, 0.66066, 0.66166, 0.66266, 0.66366, 0.66466, 0.66567, 0.66667, 0.66767, 0.66867, 0.66967, 0.67067, 0.67167,
0.67267, 0.67367, 0.67467, 0.67568, 0.67668, 0.67768, 0.67868, 0.67968, 0.68068, 0.68168, 0.68268, 0.68368, 0.68468, 0.68569, 0.68669, 0.68769, 0.68869, 0.68969, 0.69069, 0.69169, 0.69269, 0.69369, 0.69469, 0.6957,
0.6967, 0.6977, 0.6987, 0.6997, 0.7007, 0.7017, 0.7027, 0.7037, 0.7047, 0.70571, 0.70671, 0.70771, 0.70871, 0.70971, 0.71071, 0.71171, 0.71271, 0.71371, 0.71471, 0.71572, 0.71672, 0.71772, 0.71872, 0.71972,
0.72072, 0.72172, 0.72272, 0.72372, 0.72472, 0.72573, 0.72673, 0.72773, 0.72873, 0.72973, 0.73073, 0.73173, 0.73273, 0.73373, 0.73473, 0.73574, 0.73674, 0.73774, 0.73874, 0.73974, 0.74074, 0.74174, 0.74274, 0.74374,
0.74474, 0.74575, 0.74675, 0.74775, 0.74875, 0.74975, 0.75075, 0.75175, 0.75275, 0.75375, 0.75475, 0.75576, 0.75676, 0.75776, 0.75876, 0.75976, 0.76076, 0.76176, 0.76276, 0.76376, 0.76476, 0.76577, 0.76677, 0.76777,
0.76877, 0.76977, 0.77077, 0.77177, 0.77277, 0.77377, 0.77477, 0.77578, 0.77678, 0.77778, 0.77878, 0.77978, 0.78078, 0.78178, 0.78278, 0.78378, 0.78478, 0.78579, 0.78679, 0.78779, 0.78879, 0.78979, 0.79079, 0.79179,
0.79279, 0.79379, 0.79479, 0.7958, 0.7968, 0.7978, 0.7988, 0.7998, 0.8008, 0.8018, 0.8028, 0.8038, 0.8048, 0.80581, 0.80681, 0.80781, 0.80881, 0.80981, 0.81081, 0.81181, 0.81281, 0.81381, 0.81481, 0.81582,
0.81682, 0.81782, 0.81882, 0.81982, 0.82082, 0.82182, 0.82282, 0.82382, 0.82482, 0.82583, 0.82683, 0.82783, 0.82883, 0.82983, 0.83083, 0.83183, 0.83283, 0.83383, 0.83483, 0.83584, 0.83684, 0.83784, 0.83884, 0.83984,
0.84084, 0.84184, 0.84284, 0.84384, 0.84484, 0.84585, 0.84685, 0.84785, 0.84885, 0.84985, 0.85085, 0.85185, 0.85285, 0.85385, 0.85485, 0.85586, 0.85686, 0.85786, 0.85886, 0.85986, 0.86086, 0.86186, 0.86286, 0.86386,
0.86486, 0.86587, 0.86687, 0.86787, 0.86887, 0.86987, 0.87087, 0.87187, 0.87287, 0.87387, 0.87487, 0.87588, 0.87688, 0.87788, 0.87888, 0.87988, 0.88088, 0.88188, 0.88288, 0.88388, 0.88488, 0.88589, 0.88689, 0.88789,
0.88889, 0.88989, 0.89089, 0.89189, 0.89289, 0.89389, 0.89489, 0.8959, 0.8969, 0.8979, 0.8989, 0.8999, 0.9009, 0.9019, 0.9029, 0.9039, 0.9049, 0.90591, 0.90691, 0.90791, 0.90891, 0.90991, 0.91091, 0.91191,
0.91291, 0.91391, 0.91491, 0.91592, 0.91692, 0.91792, 0.91892, 0.91992, 0.92092, 0.92192, 0.92292, 0.92392, 0.92492, 0.92593, 0.92693, 0.92793, 0.92893, 0.92993, 0.93093, 0.93193, 0.93293, 0.93393, 0.93493, 0.93594,
0.93694, 0.93794, 0.93894, 0.93994, 0.94094, 0.94194, 0.94294, 0.94394, 0.94494, 0.94595, 0.94695, 0.94795, 0.94895, 0.94995, 0.95095, 0.95195, 0.95295, 0.95395, 0.95495, 0.95596, 0.95696, 0.95796, 0.95896, 0.95996,
0.96096, 0.96196, 0.96296, 0.96396, 0.96496, 0.96597, 0.96697, 0.96797, 0.96897, 0.96997, 0.97097, 0.97197, 0.97297, 0.97397, 0.97497, 0.97598, 0.97698, 0.97798, 0.97898, 0.97998, 0.98098, 0.98198, 0.98298, 0.98398,
0.98498, 0.98599, 0.98699, 0.98799, 0.98899, 0.98999, 0.99099, 0.99199, 0.99299, 0.99399, 0.99499, 0.996, 0.997, 0.998, 0.999, 1]), array([[ 0.33333, 0.33333, 0.36746, ..., 0, 0, 0],
[ 0.56667, 0.56667, 0.61635, ..., 0, 0, 0]]), 'Confidence', 'F1'], [array([ 0, 0.001001, 0.002002, 0.003003, 0.004004, 0.005005, 0.006006, 0.007007, 0.008008, 0.009009, 0.01001, 0.011011, 0.012012, 0.013013, 0.014014, 0.015015, 0.016016, 0.017017, 0.018018, 0.019019, 0.02002, 0.021021, 0.022022, 0.023023,
0.024024, 0.025025, 0.026026, 0.027027, 0.028028, 0.029029, 0.03003, 0.031031, 0.032032, 0.033033, 0.034034, 0.035035, 0.036036, 0.037037, 0.038038, 0.039039, 0.04004, 0.041041, 0.042042, 0.043043, 0.044044, 0.045045, 0.046046, 0.047047,
0.048048, 0.049049, 0.05005, 0.051051, 0.052052, 0.053053, 0.054054, 0.055055, 0.056056, 0.057057, 0.058058, 0.059059, 0.06006, 0.061061, 0.062062, 0.063063, 0.064064, 0.065065, 0.066066, 0.067067, 0.068068, 0.069069, 0.07007, 0.071071,
0.072072, 0.073073, 0.074074, 0.075075, 0.076076, 0.077077, 0.078078, 0.079079, 0.08008, 0.081081, 0.082082, 0.083083, 0.084084, 0.085085, 0.086086, 0.087087, 0.088088, 0.089089, 0.09009, 0.091091, 0.092092, 0.093093, 0.094094, 0.095095,
0.096096, 0.097097, 0.098098, 0.099099, 0.1001, 0.1011, 0.1021, 0.1031, 0.1041, 0.10511, 0.10611, 0.10711, 0.10811, 0.10911, 0.11011, 0.11111, 0.11211, 0.11311, 0.11411, 0.11512, 0.11612, 0.11712, 0.11812, 0.11912,
0.12012, 0.12112, 0.12212, 0.12312, 0.12412, 0.12513, 0.12613, 0.12713, 0.12813, 0.12913, 0.13013, 0.13113, 0.13213, 0.13313, 0.13413, 0.13514, 0.13614, 0.13714, 0.13814, 0.13914, 0.14014, 0.14114, 0.14214, 0.14314,
0.14414, 0.14515, 0.14615, 0.14715, 0.14815, 0.14915, 0.15015, 0.15115, 0.15215, 0.15315, 0.15415, 0.15516, 0.15616, 0.15716, 0.15816, 0.15916, 0.16016, 0.16116, 0.16216, 0.16316, 0.16416, 0.16517, 0.16617, 0.16717,
0.16817, 0.16917, 0.17017, 0.17117, 0.17217, 0.17317, 0.17417, 0.17518, 0.17618, 0.17718, 0.17818, 0.17918, 0.18018, 0.18118, 0.18218, 0.18318, 0.18418, 0.18519, 0.18619, 0.18719, 0.18819, 0.18919, 0.19019, 0.19119,
0.19219, 0.19319, 0.19419, 0.1952, 0.1962, 0.1972, 0.1982, 0.1992, 0.2002, 0.2012, 0.2022, 0.2032, 0.2042, 0.20521, 0.20621, 0.20721, 0.20821, 0.20921, 0.21021, 0.21121, 0.21221, 0.21321, 0.21421, 0.21522,
0.21622, 0.21722, 0.21822, 0.21922, 0.22022, 0.22122, 0.22222, 0.22322, 0.22422, 0.22523, 0.22623, 0.22723, 0.22823, 0.22923, 0.23023, 0.23123, 0.23223, 0.23323, 0.23423, 0.23524, 0.23624, 0.23724, 0.23824, 0.23924,
0.24024, 0.24124, 0.24224, 0.24324, 0.24424, 0.24525, 0.24625, 0.24725, 0.24825, 0.24925, 0.25025, 0.25125, 0.25225, 0.25325, 0.25425, 0.25526, 0.25626, 0.25726, 0.25826, 0.25926, 0.26026, 0.26126, 0.26226, 0.26326,
0.26426, 0.26527, 0.26627, 0.26727, 0.26827, 0.26927, 0.27027, 0.27127, 0.27227, 0.27327, 0.27427, 0.27528, 0.27628, 0.27728, 0.27828, 0.27928, 0.28028, 0.28128, 0.28228, 0.28328, 0.28428, 0.28529, 0.28629, 0.28729,
0.28829, 0.28929, 0.29029, 0.29129, 0.29229, 0.29329, 0.29429, 0.2953, 0.2963, 0.2973, 0.2983, 0.2993, 0.3003, 0.3013, 0.3023, 0.3033, 0.3043, 0.30531, 0.30631, 0.30731, 0.30831, 0.30931, 0.31031, 0.31131,
0.31231, 0.31331, 0.31431, 0.31532, 0.31632, 0.31732, 0.31832, 0.31932, 0.32032, 0.32132, 0.32232, 0.32332, 0.32432, 0.32533, 0.32633, 0.32733, 0.32833, 0.32933, 0.33033, 0.33133, 0.33233, 0.33333, 0.33433, 0.33534,
0.33634, 0.33734, 0.33834, 0.33934, 0.34034, 0.34134, 0.34234, 0.34334, 0.34434, 0.34535, 0.34635, 0.34735, 0.34835, 0.34935, 0.35035, 0.35135, 0.35235, 0.35335, 0.35435, 0.35536, 0.35636, 0.35736, 0.35836, 0.35936,
0.36036, 0.36136, 0.36236, 0.36336, 0.36436, 0.36537, 0.36637, 0.36737, 0.36837, 0.36937, 0.37037, 0.37137, 0.37237, 0.37337, 0.37437, 0.37538, 0.37638, 0.37738, 0.37838, 0.37938, 0.38038, 0.38138, 0.38238, 0.38338,
0.38438, 0.38539, 0.38639, 0.38739, 0.38839, 0.38939, 0.39039, 0.39139, 0.39239, 0.39339, 0.39439, 0.3954, 0.3964, 0.3974, 0.3984, 0.3994, 0.4004, 0.4014, 0.4024, 0.4034, 0.4044, 0.40541, 0.40641, 0.40741,
0.40841, 0.40941, 0.41041, 0.41141, 0.41241, 0.41341, 0.41441, 0.41542, 0.41642, 0.41742, 0.41842, 0.41942, 0.42042, 0.42142, 0.42242, 0.42342, 0.42442, 0.42543, 0.42643, 0.42743, 0.42843, 0.42943, 0.43043, 0.43143,
0.43243, 0.43343, 0.43443, 0.43544, 0.43644, 0.43744, 0.43844, 0.43944, 0.44044, 0.44144, 0.44244, 0.44344, 0.44444, 0.44545, 0.44645, 0.44745, 0.44845, 0.44945, 0.45045, 0.45145, 0.45245, 0.45345, 0.45445, 0.45546,
0.45646, 0.45746, 0.45846, 0.45946, 0.46046, 0.46146, 0.46246, 0.46346, 0.46446, 0.46547, 0.46647, 0.46747, 0.46847, 0.46947, 0.47047, 0.47147, 0.47247, 0.47347, 0.47447, 0.47548, 0.47648, 0.47748, 0.47848, 0.47948,
0.48048, 0.48148, 0.48248, 0.48348, 0.48448, 0.48549, 0.48649, 0.48749, 0.48849, 0.48949, 0.49049, 0.49149, 0.49249, 0.49349, 0.49449, 0.4955, 0.4965, 0.4975, 0.4985, 0.4995, 0.5005, 0.5015, 0.5025, 0.5035,
0.5045, 0.50551, 0.50651, 0.50751, 0.50851, 0.50951, 0.51051, 0.51151, 0.51251, 0.51351, 0.51451, 0.51552, 0.51652, 0.51752, 0.51852, 0.51952, 0.52052, 0.52152, 0.52252, 0.52352, 0.52452, 0.52553, 0.52653, 0.52753,
0.52853, 0.52953, 0.53053, 0.53153, 0.53253, 0.53353, 0.53453, 0.53554, 0.53654, 0.53754, 0.53854, 0.53954, 0.54054, 0.54154, 0.54254, 0.54354, 0.54454, 0.54555, 0.54655, 0.54755, 0.54855, 0.54955, 0.55055, 0.55155,
0.55255, 0.55355, 0.55455, 0.55556, 0.55656, 0.55756, 0.55856, 0.55956, 0.56056, 0.56156, 0.56256, 0.56356, 0.56456, 0.56557, 0.56657, 0.56757, 0.56857, 0.56957, 0.57057, 0.57157, 0.57257, 0.57357, 0.57457, 0.57558,
0.57658, 0.57758, 0.57858, 0.57958, 0.58058, 0.58158, 0.58258, 0.58358, 0.58458, 0.58559, 0.58659, 0.58759, 0.58859, 0.58959, 0.59059, 0.59159, 0.59259, 0.59359, 0.59459, 0.5956, 0.5966, 0.5976, 0.5986, 0.5996,
0.6006, 0.6016, 0.6026, 0.6036, 0.6046, 0.60561, 0.60661, 0.60761, 0.60861, 0.60961, 0.61061, 0.61161, 0.61261, 0.61361, 0.61461, 0.61562, 0.61662, 0.61762, 0.61862, 0.61962, 0.62062, 0.62162, 0.62262, 0.62362,
0.62462, 0.62563, 0.62663, 0.62763, 0.62863, 0.62963, 0.63063, 0.63163, 0.63263, 0.63363, 0.63463, 0.63564, 0.63664, 0.63764, 0.63864, 0.63964, 0.64064, 0.64164, 0.64264, 0.64364, 0.64464, 0.64565, 0.64665, 0.64765,
0.64865, 0.64965, 0.65065, 0.65165, 0.65265, 0.65365, 0.65465, 0.65566, 0.65666, 0.65766, 0.65866, 0.65966, 0.66066, 0.66166, 0.66266, 0.66366, 0.66466, 0.66567, 0.66667, 0.66767, 0.66867, 0.66967, 0.67067, 0.67167,
0.67267, 0.67367, 0.67467, 0.67568, 0.67668, 0.67768, 0.67868, 0.67968, 0.68068, 0.68168, 0.68268, 0.68368, 0.68468, 0.68569, 0.68669, 0.68769, 0.68869, 0.68969, 0.69069, 0.69169, 0.69269, 0.69369, 0.69469, 0.6957,
0.6967, 0.6977, 0.6987, 0.6997, 0.7007, 0.7017, 0.7027, 0.7037, 0.7047, 0.70571, 0.70671, 0.70771, 0.70871, 0.70971, 0.71071, 0.71171, 0.71271, 0.71371, 0.71471, 0.71572, 0.71672, 0.71772, 0.71872, 0.71972,
0.72072, 0.72172, 0.72272, 0.72372, 0.72472, 0.72573, 0.72673, 0.72773, 0.72873, 0.72973, 0.73073, 0.73173, 0.73273, 0.73373, 0.73473, 0.73574, 0.73674, 0.73774, 0.73874, 0.73974, 0.74074, 0.74174, 0.74274, 0.74374,
0.74474, 0.74575, 0.74675, 0.74775, 0.74875, 0.74975, 0.75075, 0.75175, 0.75275, 0.75375, 0.75475, 0.75576, 0.75676, 0.75776, 0.75876, 0.75976, 0.76076, 0.76176, 0.76276, 0.76376, 0.76476, 0.76577, 0.76677, 0.76777,
0.76877, 0.76977, 0.77077, 0.77177, 0.77277, 0.77377, 0.77477, 0.77578, 0.77678, 0.77778, 0.77878, 0.77978, 0.78078, 0.78178, 0.78278, 0.78378, 0.78478, 0.78579, 0.78679, 0.78779, 0.78879, 0.78979, 0.79079, 0.79179,
0.79279, 0.79379, 0.79479, 0.7958, 0.7968, 0.7978, 0.7988, 0.7998, 0.8008, 0.8018, 0.8028, 0.8038, 0.8048, 0.80581, 0.80681, 0.80781, 0.80881, 0.80981, 0.81081, 0.81181, 0.81281, 0.81381, 0.81481, 0.81582,
0.81682, 0.81782, 0.81882, 0.81982, 0.82082, 0.82182, 0.82282, 0.82382, 0.82482, 0.82583, 0.82683, 0.82783, 0.82883, 0.82983, 0.83083, 0.83183, 0.83283, 0.83383, 0.83483, 0.83584, 0.83684, 0.83784, 0.83884, 0.83984,
0.84084, 0.84184, 0.84284, 0.84384, 0.84484, 0.84585, 0.84685, 0.84785, 0.84885, 0.84985, 0.85085, 0.85185, 0.85285, 0.85385, 0.85485, 0.85586, 0.85686, 0.85786, 0.85886, 0.85986, 0.86086, 0.86186, 0.86286, 0.86386,
0.86486, 0.86587, 0.86687, 0.86787, 0.86887, 0.86987, 0.87087, 0.87187, 0.87287, 0.87387, 0.87487, 0.87588, 0.87688, 0.87788, 0.87888, 0.87988, 0.88088, 0.88188, 0.88288, 0.88388, 0.88488, 0.88589, 0.88689, 0.88789,
0.88889, 0.88989, 0.89089, 0.89189, 0.89289, 0.89389, 0.89489, 0.8959, 0.8969, 0.8979, 0.8989, 0.8999, 0.9009, 0.9019, 0.9029, 0.9039, 0.9049, 0.90591, 0.90691, 0.90791, 0.90891, 0.90991, 0.91091, 0.91191,
0.91291, 0.91391, 0.91491, 0.91592, 0.91692, 0.91792, 0.91892, 0.91992, 0.92092, 0.92192, 0.92292, 0.92392, 0.92492, 0.92593, 0.92693, 0.92793, 0.92893, 0.92993, 0.93093, 0.93193, 0.93293, 0.93393, 0.93493, 0.93594,
0.93694, 0.93794, 0.93894, 0.93994, 0.94094, 0.94194, 0.94294, 0.94394, 0.94494, 0.94595, 0.94695, 0.94795, 0.94895, 0.94995, 0.95095, 0.95195, 0.95295, 0.95395, 0.95495, 0.95596, 0.95696, 0.95796, 0.95896, 0.95996,
0.96096, 0.96196, 0.96296, 0.96396, 0.96496, 0.96597, 0.96697, 0.96797, 0.96897, 0.96997, 0.97097, 0.97197, 0.97297, 0.97397, 0.97497, 0.97598, 0.97698, 0.97798, 0.97898, 0.97998, 0.98098, 0.98198, 0.98298, 0.98398,
0.98498, 0.98599, 0.98699, 0.98799, 0.98899, 0.98999, 0.99099, 0.99199, 0.99299, 0.99399, 0.99499, 0.996, 0.997, 0.998, 0.999, 1]), array([[ 0.2, 0.2, 0.22509, ..., 1, 1, 1],
[ 0.39535, 0.39535, 0.44546, ..., 1, 1, 1]]), 'Confidence', 'Precision'], [array([ 0, 0.001001, 0.002002, 0.003003, 0.004004, 0.005005, 0.006006, 0.007007, 0.008008, 0.009009, 0.01001, 0.011011, 0.012012, 0.013013, 0.014014, 0.015015, 0.016016, 0.017017, 0.018018, 0.019019, 0.02002, 0.021021, 0.022022, 0.023023,
0.024024, 0.025025, 0.026026, 0.027027, 0.028028, 0.029029, 0.03003, 0.031031, 0.032032, 0.033033, 0.034034, 0.035035, 0.036036, 0.037037, 0.038038, 0.039039, 0.04004, 0.041041, 0.042042, 0.043043, 0.044044, 0.045045, 0.046046, 0.047047,
0.048048, 0.049049, 0.05005, 0.051051, 0.052052, 0.053053, 0.054054, 0.055055, 0.056056, 0.057057, 0.058058, 0.059059, 0.06006, 0.061061, 0.062062, 0.063063, 0.064064, 0.065065, 0.066066, 0.067067, 0.068068, 0.069069, 0.07007, 0.071071,
0.072072, 0.073073, 0.074074, 0.075075, 0.076076, 0.077077, 0.078078, 0.079079, 0.08008, 0.081081, 0.082082, 0.083083, 0.084084, 0.085085, 0.086086, 0.087087, 0.088088, 0.089089, 0.09009, 0.091091, 0.092092, 0.093093, 0.094094, 0.095095,
0.096096, 0.097097, 0.098098, 0.099099, 0.1001, 0.1011, 0.1021, 0.1031, 0.1041, 0.10511, 0.10611, 0.10711, 0.10811, 0.10911, 0.11011, 0.11111, 0.11211, 0.11311, 0.11411, 0.11512, 0.11612, 0.11712, 0.11812, 0.11912,
0.12012, 0.12112, 0.12212, 0.12312, 0.12412, 0.12513, 0.12613, 0.12713, 0.12813, 0.12913, 0.13013, 0.13113, 0.13213, 0.13313, 0.13413, 0.13514, 0.13614, 0.13714, 0.13814, 0.13914, 0.14014, 0.14114, 0.14214, 0.14314,
0.14414, 0.14515, 0.14615, 0.14715, 0.14815, 0.14915, 0.15015, 0.15115, 0.15215, 0.15315, 0.15415, 0.15516, 0.15616, 0.15716, 0.15816, 0.15916, 0.16016, 0.16116, 0.16216, 0.16316, 0.16416, 0.16517, 0.16617, 0.16717,
0.16817, 0.16917, 0.17017, 0.17117, 0.17217, 0.17317, 0.17417, 0.17518, 0.17618, 0.17718, 0.17818, 0.17918, 0.18018, 0.18118, 0.18218, 0.18318, 0.18418, 0.18519, 0.18619, 0.18719, 0.18819, 0.18919, 0.19019, 0.19119,
0.19219, 0.19319, 0.19419, 0.1952, 0.1962, 0.1972, 0.1982, 0.1992, 0.2002, 0.2012, 0.2022, 0.2032, 0.2042, 0.20521, 0.20621, 0.20721, 0.20821, 0.20921, 0.21021, 0.21121, 0.21221, 0.21321, 0.21421, 0.21522,
0.21622, 0.21722, 0.21822, 0.21922, 0.22022, 0.22122, 0.22222, 0.22322, 0.22422, 0.22523, 0.22623, 0.22723, 0.22823, 0.22923, 0.23023, 0.23123, 0.23223, 0.23323, 0.23423, 0.23524, 0.23624, 0.23724, 0.23824, 0.23924,
0.24024, 0.24124, 0.24224, 0.24324, 0.24424, 0.24525, 0.24625, 0.24725, 0.24825, 0.24925, 0.25025, 0.25125, 0.25225, 0.25325, 0.25425, 0.25526, 0.25626, 0.25726, 0.25826, 0.25926, 0.26026, 0.26126, 0.26226, 0.26326,
0.26426, 0.26527, 0.26627, 0.26727, 0.26827, 0.26927, 0.27027, 0.27127, 0.27227, 0.27327, 0.27427, 0.27528, 0.27628, 0.27728, 0.27828, 0.27928, 0.28028, 0.28128, 0.28228, 0.28328, 0.28428, 0.28529, 0.28629, 0.28729,
0.28829, 0.28929, 0.29029, 0.29129, 0.29229, 0.29329, 0.29429, 0.2953, 0.2963, 0.2973, 0.2983, 0.2993, 0.3003, 0.3013, 0.3023, 0.3033, 0.3043, 0.30531, 0.30631, 0.30731, 0.30831, 0.30931, 0.31031, 0.31131,
0.31231, 0.31331, 0.31431, 0.31532, 0.31632, 0.31732, 0.31832, 0.31932, 0.32032, 0.32132, 0.32232, 0.32332, 0.32432, 0.32533, 0.32633, 0.32733, 0.32833, 0.32933, 0.33033, 0.33133, 0.33233, 0.33333, 0.33433, 0.33534,
0.33634, 0.33734, 0.33834, 0.33934, 0.34034, 0.34134, 0.34234, 0.34334, 0.34434, 0.34535, 0.34635, 0.34735, 0.34835, 0.34935, 0.35035, 0.35135, 0.35235, 0.35335, 0.35435, 0.35536, 0.35636, 0.35736, 0.35836, 0.35936,
0.36036, 0.36136, 0.36236, 0.36336, 0.36436, 0.36537, 0.36637, 0.36737, 0.36837, 0.36937, 0.37037, 0.37137, 0.37237, 0.37337, 0.37437, 0.37538, 0.37638, 0.37738, 0.37838, 0.37938, 0.38038, 0.38138, 0.38238, 0.38338,
0.38438, 0.38539, 0.38639, 0.38739, 0.38839, 0.38939, 0.39039, 0.39139, 0.39239, 0.39339, 0.39439, 0.3954, 0.3964, 0.3974, 0.3984, 0.3994, 0.4004, 0.4014, 0.4024, 0.4034, 0.4044, 0.40541, 0.40641, 0.40741,
0.40841, 0.40941, 0.41041, 0.41141, 0.41241, 0.41341, 0.41441, 0.41542, 0.41642, 0.41742, 0.41842, 0.41942, 0.42042, 0.42142, 0.42242, 0.42342, 0.42442, 0.42543, 0.42643, 0.42743, 0.42843, 0.42943, 0.43043, 0.43143,
0.43243, 0.43343, 0.43443, 0.43544, 0.43644, 0.43744, 0.43844, 0.43944, 0.44044, 0.44144, 0.44244, 0.44344, 0.44444, 0.44545, 0.44645, 0.44745, 0.44845, 0.44945, 0.45045, 0.45145, 0.45245, 0.45345, 0.45445, 0.45546,
0.45646, 0.45746, 0.45846, 0.45946, 0.46046, 0.46146, 0.46246, 0.46346, 0.46446, 0.46547, 0.46647, 0.46747, 0.46847, 0.46947, 0.47047, 0.47147, 0.47247, 0.47347, 0.47447, 0.47548, 0.47648, 0.47748, 0.47848, 0.47948,
0.48048, 0.48148, 0.48248, 0.48348, 0.48448, 0.48549, 0.48649, 0.48749, 0.48849, 0.48949, 0.49049, 0.49149, 0.49249, 0.49349, 0.49449, 0.4955, 0.4965, 0.4975, 0.4985, 0.4995, 0.5005, 0.5015, 0.5025, 0.5035,
0.5045, 0.50551, 0.50651, 0.50751, 0.50851, 0.50951, 0.51051, 0.51151, 0.51251, 0.51351, 0.51451, 0.51552, 0.51652, 0.51752, 0.51852, 0.51952, 0.52052, 0.52152, 0.52252, 0.52352, 0.52452, 0.52553, 0.52653, 0.52753,
0.52853, 0.52953, 0.53053, 0.53153, 0.53253, 0.53353, 0.53453, 0.53554, 0.53654, 0.53754, 0.53854, 0.53954, 0.54054, 0.54154, 0.54254, 0.54354, 0.54454, 0.54555, 0.54655, 0.54755, 0.54855, 0.54955, 0.55055, 0.55155,
0.55255, 0.55355, 0.55455, 0.55556, 0.55656, 0.55756, 0.55856, 0.55956, 0.56056, 0.56156, 0.56256, 0.56356, 0.56456, 0.56557, 0.56657, 0.56757, 0.56857, 0.56957, 0.57057, 0.57157, 0.57257, 0.57357, 0.57457, 0.57558,
0.57658, 0.57758, 0.57858, 0.57958, 0.58058, 0.58158, 0.58258, 0.58358, 0.58458, 0.58559, 0.58659, 0.58759, 0.58859, 0.58959, 0.59059, 0.59159, 0.59259, 0.59359, 0.59459, 0.5956, 0.5966, 0.5976, 0.5986, 0.5996,
0.6006, 0.6016, 0.6026, 0.6036, 0.6046, 0.60561, 0.60661, 0.60761, 0.60861, 0.60961, 0.61061, 0.61161, 0.61261, 0.61361, 0.61461, 0.61562, 0.61662, 0.61762, 0.61862, 0.61962, 0.62062, 0.62162, 0.62262, 0.62362,
0.62462, 0.62563, 0.62663, 0.62763, 0.62863, 0.62963, 0.63063, 0.63163, 0.63263, 0.63363, 0.63463, 0.63564, 0.63664, 0.63764, 0.63864, 0.63964, 0.64064, 0.64164, 0.64264, 0.64364, 0.64464, 0.64565, 0.64665, 0.64765,
0.64865, 0.64965, 0.65065, 0.65165, 0.65265, 0.65365, 0.65465, 0.65566, 0.65666, 0.65766, 0.65866, 0.65966, 0.66066, 0.66166, 0.66266, 0.66366, 0.66466, 0.66567, 0.66667, 0.66767, 0.66867, 0.66967, 0.67067, 0.67167,
0.67267, 0.67367, 0.67467, 0.67568, 0.67668, 0.67768, 0.67868, 0.67968, 0.68068, 0.68168, 0.68268, 0.68368, 0.68468, 0.68569, 0.68669, 0.68769, 0.68869, 0.68969, 0.69069, 0.69169, 0.69269, 0.69369, 0.69469, 0.6957,
0.6967, 0.6977, 0.6987, 0.6997, 0.7007, 0.7017, 0.7027, 0.7037, 0.7047, 0.70571, 0.70671, 0.70771, 0.70871, 0.70971, 0.71071, 0.71171, 0.71271, 0.71371, 0.71471, 0.71572, 0.71672, 0.71772, 0.71872, 0.71972,
0.72072, 0.72172, 0.72272, 0.72372, 0.72472, 0.72573, 0.72673, 0.72773, 0.72873, 0.72973, 0.73073, 0.73173, 0.73273, 0.73373, 0.73473, 0.73574, 0.73674, 0.73774, 0.73874, 0.73974, 0.74074, 0.74174, 0.74274, 0.74374,
0.74474, 0.74575, 0.74675, 0.74775, 0.74875, 0.74975, 0.75075, 0.75175, 0.75275, 0.75375, 0.75475, 0.75576, 0.75676, 0.75776, 0.75876, 0.75976, 0.76076, 0.76176, 0.76276, 0.76376, 0.76476, 0.76577, 0.76677, 0.76777,
0.76877, 0.76977, 0.77077, 0.77177, 0.77277, 0.77377, 0.77477, 0.77578, 0.77678, 0.77778, 0.77878, 0.77978, 0.78078, 0.78178, 0.78278, 0.78378, 0.78478, 0.78579, 0.78679, 0.78779, 0.78879, 0.78979, 0.79079, 0.79179,
0.79279, 0.79379, 0.79479, 0.7958, 0.7968, 0.7978, 0.7988, 0.7998, 0.8008, 0.8018, 0.8028, 0.8038, 0.8048, 0.80581, 0.80681, 0.80781, 0.80881, 0.80981, 0.81081, 0.81181, 0.81281, 0.81381, 0.81481, 0.81582,
0.81682, 0.81782, 0.81882, 0.81982, 0.82082, 0.82182, 0.82282, 0.82382, 0.82482, 0.82583, 0.82683, 0.82783, 0.82883, 0.82983, 0.83083, 0.83183, 0.83283, 0.83383, 0.83483, 0.83584, 0.83684, 0.83784, 0.83884, 0.83984,
0.84084, 0.84184, 0.84284, 0.84384, 0.84484, 0.84585, 0.84685, 0.84785, 0.84885, 0.84985, 0.85085, 0.85185, 0.85285, 0.85385, 0.85485, 0.85586, 0.85686, 0.85786, 0.85886, 0.85986, 0.86086, 0.86186, 0.86286, 0.86386,
0.86486, 0.86587, 0.86687, 0.86787, 0.86887, 0.86987, 0.87087, 0.87187, 0.87287, 0.87387, 0.87487, 0.87588, 0.87688, 0.87788, 0.87888, 0.87988, 0.88088, 0.88188, 0.88288, 0.88388, 0.88488, 0.88589, 0.88689, 0.88789,
0.88889, 0.88989, 0.89089, 0.89189, 0.89289, 0.89389, 0.89489, 0.8959, 0.8969, 0.8979, 0.8989, 0.8999, 0.9009, 0.9019, 0.9029, 0.9039, 0.9049, 0.90591, 0.90691, 0.90791, 0.90891, 0.90991, 0.91091, 0.91191,
0.91291, 0.91391, 0.91491, 0.91592, 0.91692, 0.91792, 0.91892, 0.91992, 0.92092, 0.92192, 0.92292, 0.92392, 0.92492, 0.92593, 0.92693, 0.92793, 0.92893, 0.92993, 0.93093, 0.93193, 0.93293, 0.93393, 0.93493, 0.93594,
0.93694, 0.93794, 0.93894, 0.93994, 0.94094, 0.94194, 0.94294, 0.94394, 0.94494, 0.94595, 0.94695, 0.94795, 0.94895, 0.94995, 0.95095, 0.95195, 0.95295, 0.95395, 0.95495, 0.95596, 0.95696, 0.95796, 0.95896, 0.95996,
0.96096, 0.96196, 0.96296, 0.96396, 0.96496, 0.96597, 0.96697, 0.96797, 0.96897, 0.96997, 0.97097, 0.97197, 0.97297, 0.97397, 0.97497, 0.97598, 0.97698, 0.97798, 0.97898, 0.97998, 0.98098, 0.98198, 0.98298, 0.98398,
0.98498, 0.98599, 0.98699, 0.98799, 0.98899, 0.98999, 0.99099, 0.99199, 0.99299, 0.99399, 0.99499, 0.996, 0.997, 0.998, 0.999, 1]), array([[ 1, 1, 1, ..., 0, 0, 0],
[ 1, 1, 1, ..., 0, 0, 0]]), 'Confidence', 'Recall']]
fitness: 0.815654414412933
keys: ['metrics/precision(B)', 'metrics/recall(B)', 'metrics/mAP50(B)', 'metrics/mAP50-95(B)']
maps: array([ 0.80475, 0.7397, 0.80475, 0.86979])
names: {0: 'damagedcommercialbuilding', 1: 'damagedresidentialbuilding', 2: 'undamagedcommercialbuilding', 3: 'undamagedresidentialbuilding'}
plot: True
results_dict: {'metrics/precision(B)': 0.8003000305357673, 'metrics/recall(B)': 0.9178136342488808, 'metrics/mAP50(B)': 0.9138322182133722, 'metrics/mAP50-95(B)': 0.8047457695462175, 'fitness': 0.815654414412933}
save_dir: PosixPath('runs/detect/val')
speed: {'preprocess': 2.819538116455078, 'inference': 760.7365846633911, 'loss': 0.0054836273193359375, 'postprocess': 375.1009702682495}
task: 'detect'
In [52]:
#100 epochs is too much for batch size equal 32
#Model starts to overfit
#However YOLO automatically chooses the best iteration
figure(figsize=(15, 10), dpi=80)
# reading the image
results = img.imread('/Users/macbookair13m22022/Downloads/train11/results.png') # change this as needed
# displaying the image
plt.imshow(results)
plt.show()
In [53]:
#confution matrix of our validation dataset
figure(figsize=(20,15), dpi=80)
# reading the image
cf = img.imread('/Users/macbookair13m22022/Downloads/train11/confusion_matrix.png') # change this as needed
# displaying the image
plt.imshow(cf)
plt.show()
Submission¶
Here we just predict labels for validation dataset provided.
In [54]:
# Decoding according to the .yaml file class names order
decoding_of_predictions ={0: 'undamagedcommercialbuilding', 1: 'undamagedresidentialbuilding', 2: 'damagedresidentialbuilding', 3: 'damagedcommercialbuilding'}
directory = './Submission Data (12 images)'
# Directory to store outputs
results_directory = 'Validation_Data_Results'
# Create submission directory if it doesn't exist
if not os.path.exists(results_directory):
os.makedirs(results_directory)
# Loop through each file in the directory
for filename in os.listdir(directory):
# Check if the current object is a file and ends with .jpeg
if os.path.isfile(os.path.join(directory, filename)) and filename.lower().endswith('.jpg'):
# Perform operations on the file
file_path = os.path.join(directory, filename)
print(file_path)
print("Making a prediction on ", filename)
results = model.predict(file_path, save=True, iou=0.5, save_txt=True, conf=0.25)
for r in results:
conf_list = r.boxes.conf.numpy().tolist()
clss_list = r.boxes.cls.numpy().tolist()
original_list = clss_list
updated_list = []
for element in original_list:
updated_list.append(decoding_of_predictions[int(element)])
bounding_boxes = r.boxes.xyxy.numpy()
confidences = conf_list
class_names = updated_list
# Check if bounding boxes, confidences and class names match
if len(bounding_boxes) != len(confidences) or len(bounding_boxes) != len(class_names):
print("Error: Number of bounding boxes, confidences, and class names should be the same.")
continue
text_file_name = os.path.splitext(filename)[0]
# Creating a new .txt file for each image in the submission_directory
with open(os.path.join(results_directory, f"{text_file_name}.txt"), "w") as file:
for i in range(len(bounding_boxes)):
# Get coordinates of each bounding box
left, top, right, bottom = bounding_boxes[i]
# Write content to file in desired format
file.write(f"{class_names[i]} {confidences[i]} {left} {top} {right} {bottom}\n")
print("Output files generated successfully.")
./Submission Data (12 images)/Validation_Post_Event_009.jpg Making a prediction on Validation_Post_Event_009.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_009.jpg: 512x512 7 damagedcommercialbuildings, 8 damagedresidentialbuildings, 12 undamagedcommercialbuildings, 8 undamagedresidentialbuildings, 963.3ms Speed: 15.7ms preprocess, 963.3ms inference, 11.5ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 1 label saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_008.jpg Making a prediction on Validation_Post_Event_008.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_008.jpg: 512x512 1 damagedcommercialbuilding, 9 damagedresidentialbuildings, 1 undamagedcommercialbuilding, 11 undamagedresidentialbuildings, 743.6ms Speed: 0.8ms preprocess, 743.6ms inference, 0.5ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 2 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_011.jpg Making a prediction on Validation_Post_Event_011.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_011.jpg: 512x512 7 damagedcommercialbuildings, 7 damagedresidentialbuildings, 8 undamagedcommercialbuildings, 2 undamagedresidentialbuildings, 728.4ms Speed: 0.9ms preprocess, 728.4ms inference, 0.6ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 3 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_005.jpg Making a prediction on Validation_Post_Event_005.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_005.jpg: 512x512 3 damagedcommercialbuildings, 13 damagedresidentialbuildings, 9 undamagedcommercialbuildings, 30 undamagedresidentialbuildings, 758.1ms Speed: 1.1ms preprocess, 758.1ms inference, 0.8ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 4 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_004.jpg Making a prediction on Validation_Post_Event_004.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_004.jpg: 512x512 3 damagedcommercialbuildings, 8 damagedresidentialbuildings, 4 undamagedcommercialbuildings, 10 undamagedresidentialbuildings, 805.3ms Speed: 0.9ms preprocess, 805.3ms inference, 1.0ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 5 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_010.jpg Making a prediction on Validation_Post_Event_010.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_010.jpg: 512x512 7 damagedcommercialbuildings, 6 damagedresidentialbuildings, 9 undamagedcommercialbuildings, 5 undamagedresidentialbuildings, 834.4ms Speed: 4.1ms preprocess, 834.4ms inference, 0.7ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 6 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_006.jpg Making a prediction on Validation_Post_Event_006.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_006.jpg: 512x512 7 damagedcommercialbuildings, 14 damagedresidentialbuildings, 8 undamagedcommercialbuildings, 29 undamagedresidentialbuildings, 848.6ms Speed: 1.1ms preprocess, 848.6ms inference, 0.8ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 7 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_012.jpg Making a prediction on Validation_Post_Event_012.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_012.jpg: 512x512 2 damagedcommercialbuildings, 13 damagedresidentialbuildings, 7 undamagedcommercialbuildings, 21 undamagedresidentialbuildings, 902.1ms Speed: 0.9ms preprocess, 902.1ms inference, 0.7ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 8 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_007.jpg Making a prediction on Validation_Post_Event_007.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_007.jpg: 512x512 5 damagedcommercialbuildings, 28 damagedresidentialbuildings, 1 undamagedcommercialbuilding, 25 undamagedresidentialbuildings, 873.1ms Speed: 1.0ms preprocess, 873.1ms inference, 0.7ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 9 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_003.jpg Making a prediction on Validation_Post_Event_003.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_003.jpg: 512x512 6 damagedcommercialbuildings, 15 damagedresidentialbuildings, 1 undamagedcommercialbuilding, 8 undamagedresidentialbuildings, 755.6ms Speed: 1.0ms preprocess, 755.6ms inference, 0.8ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 10 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_002.jpg Making a prediction on Validation_Post_Event_002.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_002.jpg: 512x512 4 damagedcommercialbuildings, 12 damagedresidentialbuildings, 9 undamagedresidentialbuildings, 912.8ms Speed: 0.9ms preprocess, 912.8ms inference, 1.6ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 11 labels saved to runs/detect/predict/labels Output files generated successfully. ./Submission Data (12 images)/Validation_Post_Event_001.jpg Making a prediction on Validation_Post_Event_001.jpg image 1/1 /Users/macbookair13m22022/Downloads/Submission Data (12 images)/Validation_Post_Event_001.jpg: 512x512 1 damagedcommercialbuilding, 21 damagedresidentialbuildings, 1 undamagedcommercialbuilding, 31 undamagedresidentialbuildings, 1020.2ms Speed: 1.1ms preprocess, 1020.2ms inference, 0.7ms postprocess per image at shape (1, 3, 512, 512) Results saved to runs/detect/predict 12 labels saved to runs/detect/predict/labels Output files generated successfully.
In [55]:
# Define your source directory and the destination where the zip file will be created
source_dir = results_directory
destination_zip = 'submission'
# Create a zip file from the directory
shutil.make_archive(destination_zip, 'zip', source_dir)
print(f"Directory {source_dir} has been successfully zipped into {destination_zip}.")
Directory Validation_Data_Results has been successfully zipped into submission.